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Introduction 


Department  of  Defense  directives  and  policies  require  risk  to  be  explicitly  considered  in 
the  management  of  acquisition  programs.  In  compliance  with  these  directives,  program  managers 
conduct  risk  assessment  in  sane  form  or  another.  However,  almost  universally,  pro  ject  managers 
do  not  use  the  variety  of  quantitative  techniques  which  are  available  to  do  risk  assessment. 
Risk  assessment  is  generally  thought  of  as  an  additional  requirement  placed  on  the  program 
manager  rather  than  a  management  tool.  In  effect,  the  assessment  of  risk  and  a  statement  of  its 
expected  effect  on  program  schedule,  cost  and  performance  is  perceived  as  an  additional  hurdle 
for  the  program  manager  (PM)  to  surmount.  The  basic  problem  is  that  there  is  no  way  for  the  PM, 
who  is  new  to  risk  assessment,  to  knowledgeably  select  a  technique  or  to  easily  implement  it  once 
selected. 
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tor  this  reason,  the  Navy' s  Office  for  Acquisition  Research  tasked  Administrative  Sciences 
Corporation  to  develop  "a  management  oriented  procedure  to  simplify  the  selection  of  the  most 
appropriate  risk  assessment  technique(s)  so  that  project  managers  will  use  than  to  improve 
management  of  their  programs."  Under  this  tasking,  we  examined  the  different  techniques 
available  for  risk  assessment^  and^concluded  that  network  simulation  'is  both  cost  effective  and 
feasible  for  use  in  almost  all  acquisition  programs.  Network  simulation  has  more  general 
utility  and  applicability  than  is  commonly  believed;  it  is  the  most  flexible  technique  available 
that  is  also  analytically  rigorous.  Networks  (we  use  the  term  here  to  refer  to  both  network  and 
decision  analyses)  do  not  have  to  be  extremely  detailed  to  be  useful.  They  can  be  as  simple  or 
complicated  as  desired.  Because  the  cost  of  preparing  a  network  is  a  function  of  the  number  of 
activities  in  the  network,  a  fairly  simple  network  can  be  useful  to,  and  within  the  reach  of ,  the 
smallest  acquisition  prograns.  However,  in  the  past  risk  assessment  has  been  hard  to 
understand,  the  necessary  tools  have  not  always  being  available,  and  it  has  been  difficult  and 
expensive  to  get  started. 

Two  things  must  be  done;  1)  develop  a  network  and  2)  use  the  network  to  do  risk  assessment. 


Traditionally,  creating  the  network  has  been  a  time  consuming,  expensive  and  tedious  task  that 
appears  to  take  more  effort  than  it  is  worth.  It  certainly  has  not  been  a  job  for  the  neophyte. 
The  second  step  requires  a  network  simulator  software  package,  the  network  simulator.  (It  is 
this  second  step  that  distinguishes  risk  assessment  from  PERT  and  CP/M. )  There  are  several 
network  analysis  software  packages  around,  most  of  which  are  proprietary  and  none  of  which  are 
easy  to  use.  They  are  almost  incomprehensible  to  the  non- specialist.  These  difficulties  have 
led  to  PMs  either  passing  on  doing  quantitative  risk  assessment  or  to  use  experienced,  and 
expensive,  help  -  always  in  the  private  sector.  "Getting  started"  is  usually  the  point  where  the 
expensive  senior  analyst  enters  the  picture;  the  senior  analyst  provides  the  "getting  started" 
capability. 

As  a  result  of  -oar  critical  review  of  the  existing  procedures  and  available  software 
packages,  we  concluded  that  "front-end"  help  in  getting  started  was  necessary  -  specifically, 
assistance  in  network  creation  and  data  preparation  and  input/  We  created  the  Risk  Assessment 
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Preprocessor  (RAPP)  ^to  provide  this  help.  RAPP  is  a  canputer-based  procedure  for  simplifying 
the  process  of  creating  networks  and  preparing  the  data  for  use  with  a  network  simulator.  RAPP 
is  interactive,  easy  to  use  software  which  will  help  the  user  create  a  network  of  program 
activities  and  input  distribution  parameters.  It  does  not  actually  do  the  risk  assessment. 
RAPP's  output  is  a  data  file  which  is  input  to  one  of  the  several  currently  available  network 
simulators.  It  is  a  tool  that  makes  risk  assessment  using  Monte  Carlo  simulation  easier  and  less 
expensive  because  it  reduces  the  need  for  senior  analysts  to  structure  the  network  and  prepare 
the  data  file  for  input  to  a  risk  assessment  package.  RAPP  allows  risk  assessment  to  be  used  more 
easily  as  a  tool  for  project  management.  It  is  a  practical  tool  for  pro  ject  managers  who  want  to 

t 

use  risk  assessment. 

What  is  Risk  Assessnent? 

Risk  Assesanent  is  the  formal  method  of  providing  the  program  manager  with  quantitative 
estimates  of  the  probability  that  major  schedule  and  cost  goals,  particularly  ICC,  will  be 


reached  and  putting  bounds  on  these  probabilities.  It  includes  specifying  the  necessary 
activities  in  the  acquisition  program,  collecting  cost,  schedule  and  performance  data,  and 
generating  alternatives  in  order  to  apply  risk  reduction  and  control.  The  outputs  of  Risk 
Assessment  are  quantitative  statements  about  the  level  and  distribution  of  program  risks. 

Risk  assessment  provides  the  quantitative  analysis  necessary  for  deriving  probabilities 
for  attaining  program  objectives  based  on  a  systematic  examination  of  a  project's  activities  and 
deadlines.  With  risk  assessment,  one  can  speak  of  having  a  75%  probability  of  achieving 
scheduled  IOC.  These  probabilities  are  not  based  on  subjective,  qualitative  "feelings"  of  the 
PM,  but  instead  are  derived  frana  structured,  quantitative  analysis  of  the  program.  Thus,  they 
are  not  the  translation  of  the  high,  medium  and  low  risk  assessments  into  probabilities,  but  a 
systematic  examination  of  cost  or  schedule  objectives  based  on  a  consideration  of  activities 
required  to  meet  IOC,  the  dependence  of  activities  on  each  other,  the  time  or  cost  required  to 
complete  each  activity,  and  information  about  how  likely  (i.e.,  the  probability)  a  particular 
cost  or  schedule  estimate  is.  Risk  assessment  is  not  limited  to  a  one  time  prediction  of  the 
critical  path,  as  is  PERT  or  the  Critical  Path  Method.  A  well  structured  risk  assessment  remains 
valuable  throughout  the  program's  life  to  help  the  PM  evaluate  his  goals.  As  intermediate 
milestones  are  met,  or  not  met,  risk  assessment  can  be  used  to  re-evaluate  intervening  cost  and 
schedule  effects  and  assess  the  impact  of  these  intermediate  milestones  on  ICC.  It  can  also  be 
used  to  evaluate  the  impact  of  program  modifications.  A  program  modification  might  impact  cost 
and/or  schedule  of  several  inter-related  activities  in  different  directions.  The  value  of  a 
structured  risk  assessment  is  in  helping  the  manager  to  evaluate  the  total  effect  on  cost  and 
scheduled  ICC.  Implicit  in  such  an  assessment  is  the  ability  to  identify  and  react  to  problem 
areas  in  a  timely  way  {before  the  down  stream  effects  are  set  in  concrete). 

In  response  to  this  requirement  for  risk  assessment,  the  PM  normally  makes  a  subjective 
estimate  of  whether  the  project  is  a  high,  medium,  or  low  risk  project.  This  estimate  is  usually 
derived  by  considering  such  factors  as  whether  the  program  is  technologically  advanced,  whether 


off-the-shelf  major  components  (such  as  engines  or  avionics)  are  being  used,  and  whether  there 
are  managerially  complex  issues  involved.  Thus,  the  judgment  made  on  the  program' s  "riskiness" 
is  based  on  serious  consideration  of  major  issues.  This  judgment  is  not  translated  into 
quantitative  statements  about  meeting  milestones,  or  cost  or  performance  goals.  The  weakness 
in  this  type  of  risk  assessment  is  not  immediately  apparent.  Early  briefings  on  a  program's 
schedule  (or  cost)  appear  sound  (as  do  progress  reports  when  nothing  is  going  wrong),  but  if 
things  begin  to  go  poorly  or  if  there  are  major  changes  to  a  program,  difficulties  can  arise. 
Briefings  then  appear  to  be  based  on  wishful  thinking  or  a  lack  of  awareness  of  the  difficulties 
that  face  the  project.  Once  schedule  and  cost  problems  become  visible,  both  the  project  and  the 
PM  are  in  trouble. 

An  indirect  benefit  of  formal  risk  assessment  is  that  it  provides  structure,  and 
consequently  improved  carmunications  among  program  personnel  with  diverse  and  sometimes 
conflicting  objectives.  Quantifying  the  unknown  aspects  of  program  schedules,  costs  and 
performance  provides  a  cannon  basis  for  interpreting  the  potential  program  impacts  of  these 
unknowns,  structure  leads  to  a  better  understanding  of  the  costs  and  benefits  of  program 
decisions.  Quantification,  in  terms  of  probability  distributions  and  expected  values  of  each 
outcome,  provides  a  common  measure  of  estimated  costs  and  benefits.  This  enhances  the  program 
manager's  ability  to  assess  the  relative  importance  of  assumptions  and  expert  opinion  on  which 
schedule,  cost  and  performance  estimates  are  based. 

The  most  analytically  rigorous  method  of  risk  assessment  is  stochastic  network  analysis. 
The  most  familiar  network  analyses  for  management  are  PERT  and  CP/M.  However,  PERT  is  a 
deterministic  approach  and  is  not  a  risk  assessment  model.  Risk  assessment  requires 
probabilistic  network  analysis.  Probabilistic  network  analysis  (stochastic  network  sin  ela¬ 
tion)  is  generally  applicable  to  all  phases,  at  every  level  of  weapon  system  program  management, 
and  provides  a  choice  of  level  of  detail  for  analysis  in  program  management.  By  accounting  for 
the  impact  of  program  uncertainties  on  cost,  schedule  and  technical  performance,  probabilistic 


network  analysis  can  contribute  to  the  discovery  of  potential  problems  and  opportunities  and  the 
evaluation  of  alternative  courses  of  action.  Stochastic  network  analysis,  with  its  generality 
and  theoretical  rigor,  is  therefore  the  risk  assessment  technique  required  by  management  in 
order  to  implement  a  meaningful  risk  assessment  program. 

The  most  powerful  network  processors  incorporate  Monte  Carlo  simulation.  In  Monte  Carlo 
simulation,  parameters  are  specified  for  various  distributions  underlying  the  program  being 
studied.  Repeated  sampling  frcm  these  underlying  distributions  is  used  to  generate  estimates 
of  the  distribution  parameters  for  the  dependent  variables  of  the  model.  Several  commercial 
packages  are  available,  along  with  computer  code  required  for  their  implementation. 
Input/output,  detail  and  formats,  as  well  as  completeness,  vary,  but  all  are  based  on  the  same 
principle.  The  network  is  simulated  many  times  (1U0-1000)  and  the  outcomes  of  each  replication 
tabulated,  one  traversal  through  the  network  has  little  meaning  by  itself.  However,  many 
repetitions  will  converge  to  the  expected  value  of  the  program  outcome.  The  distribution  of 
observed  program  costs'  dispersion,  mean  and  median  provide  additional  information.  Different 
assumptions  can  be  input  and  the  simulation  run  repeated  to  see  the  impact  of  different  program 
alternatives. 

Getting  Started: 

Network  simulation  is  capable  of  being  used  routinely  by  program  management  as  an  integral 
part  of  an  existing  management  information  system  for  program  planning,  scheduling,  resource 
management,  costing  and  control.  Getting  started  is  the  hard  part  because  existing  stochastic 
network  software  packages  do  not  meet  criteria  such  as  easy  to  understand  and  perform,  output 
easily  understood  and  responsiveness. 

There  are  four  logically  distinct  parts  to  performing  risk  assessment  using  network 
simulation: 


-5- 


1)  determining  the  activities  and  events  required  and  their  order  (sequence), 

2)  collecting  and  analyzing  resource  information  (time  and  cost)  about  each  of  the 
activities  and  events, 

3)  entering  the  network  and  resource  information  into  a  network  simulation  processor 
which  will  provide  estimates  of  the  expected  schedule  times  or  costs  and  confidence 
limits  for  these,  and 

4)  manipulating  the  data  using  the  network  simulation  processor  to  generate 
management-oriented  statements  about  risk. 

For  purposes  of  developing,  maintaining  and  updating  a  risk  assessment  management  system, 
these  tasks  are  logically  separated.  RAPP/VBRT  produces  an  integrated  approach  with  the 
capability  of  separate  adjustment  to  the  component  parts. 

Network  Development.  For  risk  assessment  to  be  useful  for  program  scheduling,  resource 
allocation,  costing  and  control,  the  basic  networks  must  be  specified  at  a  level  of  detail  useful 
to  management.  Sufficient  detail  is  necessary  when  constructing  networks  so  that  the  network 
provides  realistic  projections  of  meaningful  tasks.  Too  little  detail  makes  projections  of 
questionable  usefulness.  Too  much  detail  means  excessive  data  collection  costs,  input  costs, 
computer  computation  costs  and  -  eventually  -  output  too  detailed  for  management  use  in  review 
and  control. 

When  selecting  the  events  to  be  included  in  the  networks,  one  should  consider  factors  such 
as  specific  milestones  of  interest  (e.g.,  DSARC  milestones),  appropriate  level  of  events  of 
interest  in  the  acquisition  process,  availability  of  data,  and  the  existence  of  acquisition 


program  interfaces  with  other  SYSCOMS,  Navy  laboratories,  Navy  corporate- level  management, 
other  government  agencies  and  contractors.  When  a  tentative  group  of  activities  is  identified, 
same  review  is  desirable  to  assure  accuracy  and  clarify  details. 

Data  Collection  and  Analysis.  Once  the  activities  and  events  are  established,  critically 
important  data  must  be  collected,  including  estimates  of  the  minimum,  most  likely,  and  maximum 
time  and  cost  and/or  cost  estimating  transformations  for  each  activity.  Event  data  include 
specifying  whether  any  event  is  an  origin  or  terminating  point  for  the  network,  and  the  node  logic 
associated  with  it. 

Node  logic  refers  to  the  rules  of  stochastic  network  processors  which  determine  when  and  how 
activities  are  initiated.  An  activity  exiting  frcm  a  given  node  may  be  started  upon  the 
completion  of  one  or  more  activities  or  may  be  started  according  to  same  preset  probability 
distribution.  Activities  can  also  be  initiated  based  on  the  observed  values  of  cost,  time  or 
performance  for  seme  earlier  activity  in  the  network. 

Risk  is  introduced  by  specifying  probability  distributions  for  time  of  completion,  cost  or 
performance  variables.  The  preset  distributions  are  compared  with  random  numbers  generated  by 
the  program.  The  observed  values  of  cost,  time  and  performance  are  then  generated  according  to 
the  value  of  the  observed  random  number. 

Input  Data  into  the  Network  Simulator.  The  network  and  data  on  activities  and  events  must 
be  inserted  into  the  network  simulator.  RAPP  does  this  automatically.  Data  sources  include 
existing  program  data  bases,  historical  data  from  other  programs  and  subjective  judgment  of 
experienced  personnel.  As  experience  is  gained,  a  database  should  be  developed  with 
representative  data  for  each  type  of  information  necessary. 

Do  Risk  Asaesanent.  Once  the  data  is  inserted,  the  network  processor  simulates  the  program 


many  times,  thereby  generating  useful  statistics.  Distributions  of  time,  cost  and  performance 
variables  can  be  displayed  for  each  terminal  event.  Measures  of  criticality  and  optimality  for 
various  paths  through  the  network  can  be  calculated.  Criticality  is  a  number  which  states  the 
probability  that  a  given  path  will  be  critical.  The  optimal  path  is  the  opposite  of  the  critical 
path.  The  probability  distribution  functions  generated  by  the  program  also  allow  calculation 
of  confidence  limits  (the  range  within  which  time,  cost  or  performance  can  be  expected  to  fall 
with  sane  selected  probability).  The  shape  of  the  probability  density  functions  for  a  given 
terminal  node  also  provide  intuitive  insight  concerning  uncertainty. 

Various  parameters  such  as  durations  and  costs  can  be  changed  and  the  processor  re-run  in 
order  to  determine  the  sensitivity  of  results  to  particular  assumptions.  Such  what-if  analyses 
can  be  particularly  valuable  in  determining  weak  links  or  critical  sub-elements  of  the  program' s 
schedule  and  cost  projections. 

Because  a  project  may  succeed  or  fail  in  many  ways,  each  path  can  end  in  a  different  terminal 
node.  A  histogram,  showing  the  probability  of  reaching  each  terminal  node  can  be  examined  and 
compared  with  the  (C/T/P)  distributions  for  each  terminal  node.  This  provides  valuable  insight 
regarding  possible  program  adjustments  to  activities  along  each  of  the  paths  leading  to  a 
particular  termination.  Such  information  indicates  activities  where  reallocation  of  resources 
could  improve  the  chances  of  decreased  overall  time  and  cost  or  increased  overall  performance. 

What  Does  RAPP  Do? 

RAPP  is  a  canputer-based  tool  designed  to  help  Program  Managers  and  their  immediate  staff 
create  networks  to  be  used  for  risk  assessment.  RAPP  focuses  on  the  areas  that  have  been  found  to 
be  the  most  time  consuming  and  most  expensive.  It  uses  a  series  of  prompts  specifically  tailored 
to  the  DoD's  acquisition  process  to  (1)  help  in  structuring  the  network,  (2)  insure  consistency 
of  internal  network  logic  and  (4)  provide  automated  input  to  the  risk  assessment  package. 


1.  Help  in  Structuring  the  Network.  Any  acquisition  project  has  a  series  of  objectives 
(called  milestones)  and  a  series  of  tasks  that  must  be  performed  to  meet  these  objectives.  A 
network  uses  activities  and  events  (called  arcs  and  nodes)  to  represent  this  situation.  In 
building  a  network,  a  PM  must  first  create  a  list  of  activities  and  then  order  these  in  proper 
sequence.  Proper  sequence  includes  consideration  of  predecessor  activities,  if  any,  and 
branching  from  completed  activities,  successor  activities. 

Using  a  network,  or  developing  one,  requires  the  PM  to  think  about,  and  discuss  within  the 
PMO,  the  details  of  the  program,  problem  areas  and  opportunities  to  be  exploited,  and  the  order  of 
events  and  their  dependencies  (what  activities  must  be  ccmpleted  before  others  can  begin) .  The 
amount  of  detail  in  any  network  is  a  matter  of  choice.  Any  activity  in  the  network  can  be 
expanded  into  a  subnetwork  with  subactivities.  If  a  PM  wanted  to  examine  a  particular  activity 
or  set  of  activities  more  closely,  he  could  expand  the  activities  of  interest  into  a  more  detailed 
subnetwork. 

RAPP  assists  in  this  process  by  first  asking  for  the  list  of  milestones  and  the  activities 
which  must  be  performed  to  meet  these  milestones.  The  user  has  complete  freedom  to  add  to, 
delete  from,  or  modify  this  list  as  network  development  proceeds.  Second,  RAPP  helps  the  user  by 
ordering  the  activities.  RAPP  does  this  by  asking  a  series  of  questions  about  the  milestone 
dates,  the  time  and  cost  distributions  of  predecessor  activities,  and  program  termination.  At 
any  point  in  the  process,  or  after  a  risk  assessment  has  been  performed,  the  user  can  expand  any 
activity  into  a  partitioned  subnetwork  or  aggregate  several  activities  into  one  activity  to 
develop  a  macro  network. 

2.  Insuring  Internal  Consistency.  One  of  the  major  problems  with  constructing  networks 
is  that,  with  so  many  activities,  the  human  mind  easily  loses  track  of  their  interrelationship. 
RAPP  incorporates  internal  logic  to  insure  that  activities  which  must  be  completed  or  events 


which  must  occur  before  other  activities  can  start  are  so  recorded,  and  that  resources  (time  and 
cost)  are  applied  to  each  activity. 

3.  Output  Data  Piles.  Even  with  a  completed  network,  one  cannot  automatically  do  risk 
assessment.  The  list  of  activities,  parameters  which  describe  functional  relations  between 
time  and  cost,  and  the  probability  distributions  of  time  and  resources  required  must  be  input 
into  a  network  simulator.  Previously,  this  input  was  in  a  batch  mode  and  required  a  great  deal  of 
time  consuming  data  processing  and  delay  in  receiving  the  processor  output.  RAPP  is  integrated 
with  the  network  simulator,  VERT.  Thus,  once  the  network  is  created  by  RAPP  and  the  necessary 
data  inserted  in  RAPP,  data  files  are  created  which  can  be  read  directly  into  VERT.  Internal 
RAPP  logic  insures  that  this  data  is  consistent  with  VERT  data  requirements.  This  method  of 
creating  a  network  is  a  major  departure  frcm  the  older  "batch"  mode  of  data  entry;  not  only  is  it 
faster,  it  is  also  much  easier  and  provides  a  better  response  to  project  managers. 

What  Does  VERT  Do? 

VERT  is  the  major  network  simulator  into  which  the  data,  developed  by  the  user  with  RAPP's 
help,  is  input.  This  network  simulator  addresses  schedule,  cost  and  technical  risks,  and  their 
impacts  on  a  weapon  system  acquisition  program.  The  network  simulation  should  provide  the 
project  manager  with  information  for  formulating  initial  project  scheduling  and  acquisition 
strategy  plans.  It  assists  the  user  to: 

-  develop  a  project  plan  which  meets  management's  time  objectives 

-  establish  realistic  schedules 

-  measure  the  inpact  of  alternative  decisions  on  overall  project  performance 
before  firm  policies  are  laid  down 

-  highlight  project  activities  and  problems  requiring  special  attention  early 
enough  to  allow  preventive  measures  to  be  taken. 


VERT,  like  similar  network  simulation  software  packages,  is  used  to  determine  the 
interrelated  schedule  and  cost  risks  resulting  from  time,  cost,  and  technical  performance 
uncertainties.  For  sane  efforts,  the  analysis  might  address  only  schedule  risk.  For  other 
requirements,  cost  might  be  the  only  issue.  When  used  at  its  full  capability,  the  model  serves 
other  management  purposes  such  as  project  estimating,  planning,  scheduling,  resourcing, 
costing,  and  control.  Additionally,  VERT  can  be  coupled  to  a  computer  graphics  package  to 
develop  computer  generated  networks  and  X-Y  plots. 


